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general considerations associated with mail surveys and other 
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ABSTRACT 



Designing Questionnaire Items: Lessons Learned 
from Faculty and Student Surveys 

Andrea Meld, Ph.D. 
City University, Bellevuc, WA 

Educational decisions in higher education are most often based on some form of 
evaluation, for example, the self-study conducted for accreditation review is 
an extensive example. Frequently, these evaluations take the form of faculty 
and student surveys. This paper explores general considerations such as: 1) 
hov to avoid response bias, 2) hov to reduce nonresponse, 3) reliability and 
validity in survey research, 4) the effects of different response modes, and 
other general considerations. 

The advantages and disadvantages of open vs. close^^ question forms r dis- 
cussed in some detail. "Open" questions allow respondents to stat" anions 
in their own wordfi, and may reveal important issues which the evaiudtor did 
not anticipate. Open questions provide respondents with the opportunity to 
express their thoughts and opinions, but results are more difficult to 
analyze. 

On the other hand, "closed" questions generally produce more reliable data 
which can be statistically analyzed. Responses of individuals and groups can 
be quantified and compared. Thus, question form not only determines the 
quality of information or opinion elicited, but also affects data analysis and 
interpretation. 



Designing Questionnaire Items: Lessons Learned from Faculty and Student Surveys 

Surveys can provide a useful source of information abou^ the quality of 
higher education which may not attainable by other research methods. However, 
survey reiearch has its perils and pitfalls. This paper discusses survey 
research for program and institutional evaluation such as the accreditation 
review process and outcotne assessment. Practical suggesting are offered, in 
non-technical language, for implementing survey research methods and developing 
more useful questionnaire items. For the sake of simplicity, the term "survey" 
refers to a mailed questionnaire, however, similar recommendations apply as 
well to phone and in^person interviews. 

General Considerations 

Bias in Response 

Assuming that sampling techniques are not a source of bias (Blalock, 
1979), bias can still be a problem in campus surveys because of certain human 
or social tendencies, especially if faculty and students believe (erroneously) 
that their answers can be traced back to them (e.g., "my instructor will 
recognize my handwriting"). Three major sources of oms are acquiescence, 
social desirability, and atypical responses (Fowler, 1988). 

Acquiescence is the huican tendency to agree or respond "yes" or "true" to 
questions, regardless of their content; individuals appear to vary in the 
exten- of this tendency. One way to avoid this response bias is to write half 
of the items so that indicating agreement is actually a negative evaluation: 

"Students are prepared for upper-division classes." (positive] 
"Faculty support from administration should be irr.proved." (negative; 



Social desirability bias occurs when individuals respond positively to 
items which are congruent with socially acceptable attitudes or behaviors. 
However, this source of bias is probably not a deliberate attempt to "fake" the 
survey instrument. For example, faculty members may be reluctant to admit they 
are experiencing any problems or that academic achievement of their students is 
not excellent. They may feel t!iat such admittance reflects poorly on them as 
individuals and their academic department or the institution as a whole. 
Similarly, students may experience a reluctance to admit they are not satisfied 
with programs that represent a considerable investment of their time, energy, 
and resources or that they are -^ot "good" students, which may create cognitive 
dissonance. Also, they may not wish to make an instructor or the school "look 
bafi," for various reasons, even though negative information will be used to 
imptove and upgrade programs. 

In addition, individuals tend to underreport conditions that carry a 
social stigma, such as mental illness, physical or learning disabilities, 
alcoholism, drug abuse, or embarrassing behavior. If sensitive information is 
being asked in a survey, be aware that even seemingly innocent questions may 
embarrass someone. Consider whether such questions are really necessary. 

Atypical response occur when individuals respond atypically or unusually, 
or make choices others are unlikely to make, regardless of item content. 
Faculty and students who believe that the survey is overly intrusive or even 
pointless may express their discomfort or annoyance this way. While there is 
not mach point in debating whether such individuals are "ab;*ormal" or not, one 
solution is to bet aside atypical cases, that is, those with incopgru<=^nt 
responses, and analyze them separately. Many of these response biases can be 
avoided by ensuring faculty and student respondents that their ansuers will be 
confidential or anonymous. Explain the purpose and object! es of the survey 



research and hov the information acquired vill be used to improve instructional 
programs and services, rather than used in any punitive way. Trust between 
those conducting the research and the sample of respondents is essential. 



Nonresponse 

Nonresponse can refer to particular items which see individuals choose 
not to answer. However, when faculty and students selected for your sample 
refuse to respond at all there is the much greater problem of sampling bias. 
Common reasons for the nonresponse of faculty and students include: the 
questionnaire did not reach them because they were away on vacation or changed 
their address; they refused or failed to supply information; they were unable 
to supply information. Bias occurs when response rate is low because only 
those who are interested in the issuer, return the survey. These individuals 
differ in some systematic way from those who do not return the survey. 

Ways to Reduce Nonresponse 

Questionnaires apparently work best with individuals who are highly 
literate, interested in research, and motivated to return the form. (Fowler, 
1988) Thus, the self-administered form should be veil suited for faculty and 
students. Almost anything that makes a mailed questionnaire look professional 
and personalized, such as an attractive color or letterhead, will increase 
response rates (Fowler, 1988). The instrument should be easy to complete and 
not overly long; the directions should be easy and clear; the form should be 

^tractive, easy to read, and uncluttered. Follow-Up reminders, either by 
nostcard or phone are also important ways to reduce nonresponse. 



Reliability and Validi ty 

A reliable measure is one that produces consistent results which are not 
influenced by extraneous variables. To the extent that questionnaire items are 
clearly written and unambiguous, that is, individuals reading them will 
interpret them the same way, a survey will be reliable. Internal consistency, 
another form of reliability, refers to items on the same scale (which are 
assumed to measure the same construct), being highly intercorrelated. 

Validity of a questionnaire can be more difficult to achieve and 
determine. People do not always give valid information on a survey for several 
reasons: they do not know the information; they cannot recall it; they don*t 
understand the question; or they feel uncomtortable answering the question in 
the survey context. Recently, the cognitive aspects of designing and testing 
questionnaires, such as the validity of self-report data, have been explored in 
laboratory studies (e.g., Lessler, Mitzel, Salter, Tourangeau, 1985; Jabine, 
Straf, Tanur, & Tourangeau, 1984). 

Another way to assess validity is to compare questionnaire responses with 
a different measure, such as records or test data, nieasuring a similar concept. 
However, since these correlations usually tend to run low, in the neighborhood 
of .35 (Fowler, 1988), the usefulness of this approach to validity is limited. 

"Agree/Disagree" and "Don*t Know" Responses 

'Agree/Disagree" items are commonly used in surveys. They seem to be 
simple, but should be used with care. Statements should have only one 
diiaension, and they not be double-barieled as in this example: 

"Completing my MBA will improve my personal skills and advance my career." 
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the end of continuum, rather than middle. Does disagreement with the statement 
below indicate that an individual person believes that the alumni association 
is doing an excellent job or a poor one? 

"The alumni association is doing a fair job." 

"Don't Know" answers can have different meanings. In any case, they are 
often difficult to interpret and like "missing values," can complicate data 
analysis. "Don't know" is not the same as a neutral response category on a 
scale and requires different coding procedures. If the number of "don't 
know's" is a large percentage of item responses, items may be poorly written. 

In general, if the survey topic or particular items are about feelings or 
experiences, "don't know" may indicate an unwillingness to respond, not 
uncommon with individuals who may find it threatening to appear critical cf 
themselves or an instructor. For example, "don't know" might be a frequent 
response to inti^^oive statements such as the following: 

"I find it difficult to keep up with course assig.»inents because of my 

study skills." 

"Problems in my p'^rsonal life have prevented me from attending class." 

A better approach to ite-n construction might be: 

"I would be interested in taking a study skills workshop." or 
"Time conflicts sometimes make it difficult to attend class." 

On the other hand, students and faculty may simply be unfamiliar with the 
topic. Certain questions can be screened by a preceding question to see 
whether respondents are familiar with subject and should go any further. In 
that way, those responding will at least know something about the subiect. For 
example : 

ERIC 5 



"Do you use the library's on-line reference services?" (yes/no) 
(if yes) "How would you rate the quality of this service?" 



Open vs> Closed Questions 

Open questions have certain advantages: they describe the "real" views of 
faculty and staff and permit unanticipated answers. Often, people prefer tc 
answer questions concerning issues of importance to them in their own vords. 
They open-ended approach is also more appropriate if the list oi possible 
answers to a question is very long. 

On the other hand, closed questions are more satisfactory because the task 
of responding to a given item is more reliable and they constrain the number of 
possible answers and eliminate rare answers or those not appropriate for the 
analysis. Closed questions are also better for constructing rating scales 
which yield ordinal data. Thus, data analysis and interpretation can be far 
less time-consuming and easier to automate. 

In our surveys of faculty and students, we found that the use of a closed 
questions tended co produce generally positive mean responses, whereas faculty 
and students were moru likely to use the open format to express their 
frustrations or grievances in negative terms. (Others did express positive 
feelings or omitted the open-ended questions.) Since the use of open or a 
closed questions may influence the direction of expressed attitudes, perhaps 
both approaches should be used as a check to compare group trends with the more 
extreme views of particular individuals. 



Pretesting 

Every questionnaire should be pretested, no matter how experienced the 
researcher. Once questionnaires are printed ar-^ i^ta collection starts, 
changes can be expensive and difficult. If wording is changed, already 
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completed questionnaires have cannot be included in the analysis. Pretest a 
self-administered survey by having a small group complete the instrument. The 
research^^r should then lead a group discussion and find out if the instructioris 
were clear, vhetber any questions were confusing or difficult; and if the^-e 
were any format or design problems. 

Ethical Issues 

The essential ethical principals in survey research in the academic 
setting or elsewhere are the following: 

1) there should be no adverse effects to respondents; 

2) data must be kept confidential; 

3) data analysis and utilization should bo appropriate. 

Student and faculty respondents should be informed of the department or 
organization conducting the research; the investigators' names; the purpose -)£ 
the survey research project; and how confidentiality will be protected. 
Student and faculty should be told that their cooperation is voltmtary ^and that 
refusal to participate will not have any adverse effect or them, and that they 
can skip any questions they choose not co answer. 

Constructing Items f o r Accreditation Review 
City University, a private schoo] serving adult students, is locattid in 
Bellevue, WA. In July, 1989, the author constructed a set of surveys based on 
questions in the Northwest Association of !?chcoI? ^nd Coll^^^^s Arrr^^^ i i nn 
Handbook, (1988 Edition) ^ as :t of the re-acci -ditation process. In the cas»- 
of faculty rating proposed means of improving faculty effectiveness, written 
comments from earlier surveys which formed the basis o? response categories. 
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The actual survey items were written using wording similar to the Handbook 
questions, and were reviewed by faculty other than the researcher. The surveys 
were sent to a random sample of 700 students, 475 faculty, and 700 alumni and 
former students. Response rates for the three samples were ranged from .30 to 
.40, 

The iteiiis on the followii.j pages are actual examples to illustrate how 
broad topics can be converted into objective (closed) response items. Included 
are faculty ratings of ways to improve faculty effectiveness; ratings of the 
physical plant, classrooms, and study or lounge areas; and alumni ratings of 
programs in terms of professional advancement. These items are meant to 
suggest ways gathering objective data on topics of broad concern. Clearly, 
each institution m'. .t construct items that are suitable to itj philosophy and 
unique characteristics. 
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Handbook Question .: 

"Comment on changes which .Ight be made in departmental policies and procedures 
to improve faculty effectiveness." 

Survey Item (for Faculty) : 

What do you think is the relative importance cf these items for improving 

faculty effectiveness? (l=high; ll=lov) 

RESULTS: 

1 Ensuring that entering stuc' ts meet minimum competencies in English and 
math; prerequisites for advanced classes. 

2 An increase in faculty salary. 

3 A faculty library of teaching and instructional materials, such as slides, 
overhead displays, other visual ^.ids. 

4 An ongoing review of courses, goals, and objectives with full faculty 
participation. 

5 Funding for professional faculty development, such as training, members.np 

dues, conference attendance and presentation. 

6 Better communication between administration and faculty on i^hort- and 
long-range goals for City University. 

7 Guest speakers at faculty meetings addressing teaching methods and 
academic subjects for faculty development. 

8 Increased clerical and secreta: ial support for faculty. 

9 Peer evaluation of faculty effectiveness, as well as the SIR forms. 

10 Fostering "team spirit" among faculty members to reduce turnover rate by 
organizing various group projects and events for faculty. 

11 Greater emphasis on computer applications for classes. ti.-i:2) 



Handbook Questions : 

"Evaluate how veil the physical plant serves the needs of students and staff." 
"Carefully review maintenance and housekeeping standards and attractiveness of 
the grounds. What improvements are needed, if any?" 

Sur^^ey Itezi (Faculty, Studencs^ Alumni and Former Students) : 
^""2. Hov do you feel about the classrooms in which you teach, in terras of 
lighting, spaciousness, and comfort for both instructor and student? 

I Very I Somewhat j Neither j Somevhat j Very j 

j Satisfied { Satisfied {Satisfied nor jDissatisf iedjDissatisf ied | 

j j |Dissatisf ied J j j 

I (1) ; (2) i (3) i (4) i (5) . 

29.5% 45.4% 8.1^^ 13. n 3.1% 

(n=491) 

C-3. Hov veil do the study, lounge, ant -t ^?reas in the facility where you 
teach meet the needs of faculty ai.d students? 

} Very | Somewhat | Not \ Somewhat ; Very | 

j Adequately j Adequately | Applicable j Inadequately : Inadequately; 
i (1) 1 (2) i (3) ; (4) I (5) ! 

15.5% 37.0% 13.3% 22.0% 11.6% 
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C-4. How would you rate the houseke^'ping, maintenance, and attractiveness of 
the facility where you teach? 



j Very | Somewhat | Neither j Somewhat j Very { 

j Well-Kept j Well-Kept |Well-Kept nor jPoorly Kept jPoorly Kept | 

j j I Poorly Kept j j j 

j (1) I (2) I (3) I (4) I (5) ! 

28.6% 36.9% 18.1% 12.4% 4.1% 

Handbook Question : 

"i^hat has been learned about the educational effectiveness of the various 
.nstructional programs from former students who left before completing their 
programs of study?" 

Survey Item (For Alumni and Former Students) ; 

E-4. Kow would you rate the instructional programs at City University in 
preparing you for professional work and career advancement? 

: Very { Somewhat { Neither j Somewhat I Very | 

j Effective 1 Effective j Effective norj Ineffective j Ineffective I 

j j j Ineffective j | ; 

; (1) I (2) 1 (3) ; (4) i (5) ; 

35.5% 59.1% 4.5% .9% 0.0% 

(n=110) 
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Final Pointers and Reminders 



Avoid asking for information that is redundant with other data sources 
which are accurate and easy to across. 

2. Make sure that student, faculty, and alumni respondents are sufficiently 
motivated to respond to questionnaire items. 

3. Avoid wording which encourages favorable responses, thus biasing results. 

4. Also avoid wording vrhich is ambiguous, cryptic, or loaded with jargon. 

5. Pre^test all questions. Don't include those which respondents will find 
ridiculous or trivial. 

6. The form should not be too long or too complicated to complete. 

7. Completing the form should be an enjoyable or educational experience for 
the resp'^ndent. 

8. Results should be announced and utilized to provide feedback to 
respondents, and provide a sense of closure, that is, their participation 
made a difference. When students, faculty, and others see actual results 
from surveys, they are more likely to respond to th»:;m in the future. 
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